Systems and methods for a configurable sensor system

ABSTRACT

The present disclosure relates generally to systems and methods for generating, processing and correlating data from multiple sensors in an autonomous navigation system, and more particularly to the utilization of configurable and dynamic sensor modules within light detection and ranging systems that enable an improved correlation between sensor data as well as configurability and responsiveness of the system to its surrounding environment.

BACKGROUND A. Technical Field

The present disclosure relates generally to systems and methods forgenerating, processing and correlating data from multiple sensors in anautonomous navigation system, and more particularly to the utilizationof configurable and dynamic sensor modules within light detection andranging (hereinafter, “LIDAR”) systems that enable an improvedcorrelation between sensor data as well as configurability andresponsiveness of the system to its surrounding environment.

B. Background

One skilled in the art will understand the importance in the accuracyand timely analysis of sensor data within autonomous navigation systems.Autonomous navigation requires that a computerized system receive datafrom sensors, form a sufficiently accurate representation of itsenvironment, and make decisions based on that data in real time. Anyerror in the interpretation of the sensor data or delays in timelyinitiating a responsive action to this sensor data can have undesiredconsequences. Modern autonomous systems must process data from multiplediscrete sensor systems and interpret their combined outputs. Thequantity of data from these various sensors can be immense and simplyprocessing and correlating the outputs from different sensors mayrequire a significant amount of processing power and time.

Implementations of a LiDAR or mixed-sensor system may have differentoperational requirements such as field-of-view, range, response rate,etc. Adapting these systems to different types of autonomous navigationsystems and vehicles may be challenging as the environment and intendeduse of the system can present varying performance requirements. Simplypositioning and integrating sensors within different types of vehiclesmay present problems for many of the prior-art, sensor-based navigationsystems. For example, sensors cannot be easily collocated because thedata is subject to parallax error caused by data taken from differentvantage points. In addition, these sensors may also have different ratesof data collection. Improper correlation of this sensor data may resultin motion errors or conflicts across the sensor data. Accordingly, theautonomous system must first process and interpret data from relevantsensors followed by correlating them with one another before anyautonomous navigation decisions can be made. Any unnecessary delaywithin the decision-making process may result in a failure of theautonomous driving system.

Accordingly, what is needed are systems and methods that provideconfigurable, accurate, timely and efficient solutions for the receptionand processing of sensor data across a plurality of sensors installedwithin an autonomous navigation system.

BRIEF DESCRIPTION OF THE DRAWINGS

References will be made to embodiments of the invention, examples ofwhich may be illustrated in the accompanying figures. These figures areintended to be illustrative, not limiting. Although the invention isgenerally described in the context of these embodiments, it should beunderstood that it is not intended to limit the scope of the inventionto these particular embodiments. Items in the figures are not to scale.

Figure (“FIG.”) 1 depicts the operation of a LiDAR system according toembodiments of the present document.

FIG. 2A illustrates the operation of a LiDAR system and multiple returnlight signals according to embodiments of the present document.

FIG. 2B depicts a LIDAR system with an oscillating mirror according toembodiments of the present document.

FIG. 3A depicts a distributed sensor system installed in an automobileutilizing a suite of sensors coupled to a microcontroller (hereinafter,“MCU”) according to embodiments of the present document.

FIG. 3B depicts the framework for a sensor system according toembodiments of the current disclosure.

FIG. 3C depicts the operation of an MCU in an autonomous driving systemutilizing sensor modules and a sensor bus according to embodiments ofthe current disclosure.

FIGS. 3D and 3E illustrate methods for dynamically configuring differentsensors and sensor types within an autonomous navigation systemaccording to embodiments of the current disclosure.

FIG. 3F illustrates a method for updating calibration parameters in acalibration engine according to embodiments of the current disclosure.

FIG. 4A and FIG. 4B depict configurable sensor architectures accordingto embodiments of the current disclosure.

FIG. 4C illustrates a lissajous scan pattern and resolution according toembodiments of the present disclosure. FIGS. 4D, FIG. 4E, and FIG. 4Fillustrate scan resolutions for a field of view (FOV) according toembodiments of the present disclosure.

FIG. 4G illustrates a specific scanning pattern for a sensor modulecomprising eight sensors according to embodiments of the presentdisclosure.

FIG. 4H and FIG. 4I illustrate exemplary sensor square and pie wedgeconfigurations according to embodiments of the present disclosure.

FIG. 4J illustrates a sensor system that supports detection of objectswith various sensor types including LIDAR, infrared radiation (IR),ambient light modalities to detect range, reflectivity, temperature andcolor respectively according to embodiments of the present disclosure.

FIG. 5 depicts a simplified block diagram of a computingdevice/information handling system for an automotive application, inaccordance with embodiments of the present document.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following description, for purposes of explanation, specificdetails are set forth in order to provide an understanding of theinvention. It will be apparent, however, to one skilled in the art thatthe invention can be practiced without these details. Furthermore, oneskilled in the art will recognize that embodiments of the presentinvention, described below, may be implemented in a variety of ways,such as a process, an apparatus, a system, a device, or a method on atangible computer-readable medium.

Components, or modules, shown in diagrams are illustrative of exemplaryembodiments of the invention and are meant to avoid obscuring theinvention. It shall also be understood that throughout this discussionthat components may be described as separate functional units, which maycomprise sub-units, but those skilled in the art will recognize thatvarious components, or portions thereof, may be divided into separatecomponents or may be integrated together, including integrated within asingle system or component. It should be noted that functions oroperations discussed herein may be implemented as components. Componentsmay be implemented in software, hardware, or a combination thereof.

Furthermore, connections between components or systems within thefigures are not intended to be limited to direct connections. Rather,data between these components may be modified, re-formatted, orotherwise changed by intermediary components. Also, additional or fewerconnections may be used. It shall also be noted that the terms“coupled,” “connected,” or “communicatively coupled” shall be understoodto include direct connections, indirect connections through one or moreintermediary devices, and wireless connections.

Reference in the specification to “one embodiment,” “preferredembodiment,” “an embodiment,” or “embodiments” means that a particularfeature, structure, characteristic, or function described in connectionwith the embodiment is included in at least one embodiment of theinvention and may be in more than one embodiment. Also, the appearancesof the above-noted phrases in various places in the specification arenot necessarily all referring to the same embodiment or embodiments.

The use of certain terms in various places in the specification is forillustration and should not be construed as limiting. A service,function, or resource is not limited to a single service, function, orresource; usage of these terms may refer to a grouping of relatedservices, functions, or resources, which may be distributed oraggregated.

The terms “include,” “including,” “comprise,” and “comprising” shall beunderstood to be open terms and any lists that follow are examples andnot meant to be limited to the listed items. Any headings used hereinare for organizational purposes only and shall not be used to limit thescope of the description or the claims. Each reference mentioned in thispatent document is incorporate by reference herein in its entirety.

Furthermore, one skilled in the art shall recognize that: (1) certainsteps may optionally be performed; (2) steps may not be limited to thespecific order set forth herein; (3) certain steps may be performed indifferent orders; and (4) certain steps may be done concurrently.

A light detection and ranging system, such as a LIDAR system, may be atool to measure the shape and contour of the environment surrounding thesystem. LIDAR systems may be applied to numerous applications includingboth autonomous navigation and aerial mapping of a surface. LIDARsystems emit a light pulse that is subsequently reflected off an objectwithin the environment in which a system operates. The time each pulsetravels from being emitted to being received may be measured (i.e.,time-of-flight “TOF”) to determine the distance between the object andthe LIDAR system. The science is based on the physics of light andoptics.

In a LIDAR system, light may be emitted from a rapidly firing laser.Laser light travels through a medium and reflects off points of surfacesin the environment such as buildings, tree branches and vehicles. Thereflected light energy returns to a LIDAR transceiver (detector) whereit is recorded and used to map the environment.

FIG. 1 depicts the operation of LiDAR components 102 and data analysis &interpretation 109 according to embodiments of the present disclosure.LiDAR components 102 may comprise a transmitter 104 that transmitsemitted light signal 110, receiver 106 comprising a detector, and systemcontrol and data acquisition 108. LiDAR components 102 may be referredto as a LIDAR transceiver. Emitted light signal 110 propagates through amedium and reflects off object 112. Return light signal 114 propagatesthrough the medium and is received by receiver 106. System control anddata acquisition 108 may control the light emission by transmitter 104and the data acquisition may record the return light signal 114 detectedby receiver 106. Data analysis & interpretation 109 may receive anoutput via connection 116 from system control and data acquisition 108and perform data analysis functions. Connection 116 may be implementedwith a wireless or non-contact communication method. Transmitter 104 andreceiver 106 may include optical lens and mirrors (not shown).Transmitter 104 may emit a laser beam having a plurality of pulses in aparticular sequence. In some embodiments, light detection and rangingcomponents 102 and data analysis & interpretation 109 comprise a LIDARsystem. A design element of receiver 106 is a horizontal field of view(hereinafter, “FOV”) and a vertical FOV. One skilled in the art willrecognize that the FOV effectively defines the visibility area relatingto the specific LiDAR system. The horizontal and vertical FOVs may bedefined by a single LiDAR sensor or may relate to a plurality ofconfigurable sensors (which may be exclusively LiDAR sensors or maycomprise different types of sensors). The FOV may be considered ascanning area for a LIDAR system. A scanning mirror may be utilized toobtain a scanned FOV.

FIG. 2A illustrates the operation 200 of LiDAR system 202 includingmultiple return light signals: (1) return signal 203 and (2) returnsignal 205 according to embodiments of the present document. Due to thelaser's beam divergence, a single laser firing often hits multipleobjects producing multiple returns. The light detection and rangingsystem 202 may analyze multiple returns and may report either thestrongest return, the last return, or both returns. Per FIG. 2A, lightdetection and ranging system 202 emits a laser in the direction of nearwall 204 and far wall 208. As illustrated, the majority of the beam hitsthe near wall 204 at area 206 resulting in return signal 203, andanother portion of the beam hits the far wall 208 at area 210 resultingin return signal 205. Return signal 203 may have a shorter TOF and astronger received signal strength compared with return signal 205. Inboth single and multiple return LIDAR systems, it is important that thereturn signal is accurately associated with the transmitted light signalso that an accurate TOF is calculated.

Some embodiments of a LIDAR system may capture distance data in a 2-D(i.e. single plane) point cloud manner. These LIDAR systems may be oftenused in industrial applications and may be often repurposed forsurveying, mapping, autonomous navigation, and other uses. Someembodiments of these devices rely on the use of a single laseremitter/detector pair combined with some type of moving mirror to effectscanning across at least one plane. This mirror not only reflects theemitted light from the diode but may also reflect the return light tothe detector. Use of an oscillating mirror in this application may be ameans to achieving 90-180-360 degrees of azimuth (horizontal) view whilesimplifying both the system design and manufacturability. Manyapplications require more data than just a single 2-D plane. The 2-Dpoint cloud may be expanded to form a 3-D point cloud, where multiple2-D clouds are used, each pointing at a different elevation (vertical)angle. Design elements of the receiver of light detection and rangingsystem 202 include the horizontal FOV and the vertical FOV.

FIG. 2B depicts a LIDAR system 250 with an oscillating mirror accordingto embodiments of the present document. LIDAR system 250 employs asingle laser emitter/detector combined with an oscillating mirror toeffectively scan across a plane. Distance measurements performed by sucha system are effectively two-dimensional (i.e., planar), and thecaptured distance points are rendered as a 2-D (i.e., single plane)point cloud. In some embodiments, but without limitations, oscillatingmirrors are oscillated at very fast speeds (e.g., thousands of cyclesper minute).

LIDAR system 250 comprises laser electronics 252, which comprises asingle light emitter and light detector. The emitted laser signal 251may be directed to a fixed mirror 254, which reflects the emitted lasersignal 251 to oscillating mirror 256. As oscillating mirror 256“oscillates”, the emitted laser signal 251 may reflect off object 258 inits propagation path. The reflected signal 253 may be coupled to thedetector in laser electronics 252 via the oscillating mirror 256 andfixed mirror 254. Design elements of the receiver of LIDAR system 250include the horizontal FOV and the vertical FOV, which defines ascanning area.

FIG. 3A depicts a distributed sensor system 300 installed in anautomobile utilizing a suite of sensors coupled to an MCU 302 accordingto embodiments of the present disclosure. The suite of sensors includessensor module 304, sensor module 306, sensor module 308, sensor module310 and sensor module 312. The term “sensor module” is intended to bebroadly defined and includes implementations of single sensor modulesand multi-sensor modules. In addition, the types of sensor(s) within asensor module may vary depending on the configuration of the system. Incertain instances, a sensor module may comprise a single sensor(hereinafter, “single sensor module”) such as a LiDAR sensor or multiplesensors (hereinafter, “multi-sensor module”). A multi-sensor module maycomprise a plurality of integrated sensors, a plurality of discretesensors or a combination thereof. The multi-sensor module may alsocomprise a plurality of LiDAR sensors or a plurality of different typesof sensors that are correlated within the module. As shown in FIG. 3A,the suite of sensor modules may be distributed in a variety of locationon the vehicle. Correlated sensor data from the various sensor modulesare provided to the MCU 302 for analysis and decision processing. Theconnectivity between the sensor modules and the MCU 302 is provided by asensor bus that may transmit the different sensor data in a serialmanner (there may be other embodiments in which sensor data istransmitted on a parallel bus).

As previously described, a sensor module may comprise a single sensor ormultiple sensors and support various types of sensors such as a LIDARtransceiver, thermal/far IR sensor, visible/near IR sensor or othertypes of sensor known to one of skill in the art. The sensor structuremay have various shapes including a modular design that is rectangularor a wedge shaped that may be tiled together and/or stacked and mayallow for a design that can go around corners. These different sensorshapes allow configurability of the sensor module includingconfigurability of FOV, sensor range, etc. Based on the particularconfiguration of the sensor module and corresponding FOV, different scanpatterns and resolutions may be implemented.

MCU 302 may be coupled to an Autonomous Driving System Control Unit(hereinafter, “ADSCU”) 301. In certain embodiments, the ADSCU 301 mayprovide sensor instructions and information to MCU 302.

FIG. 3B depicts the framework for a sensor system 320 according toembodiments of the current disclosure. Sensor system 322 may besupported by MCU 324 and its associated software. Sensor system 322 mayinclude scan mirror 326, ASICs 328, firmware 330 and sensors 332. Insome embodiments, scan mirror 326 may be a dual axis resonant scanningmirror. In some embodiments, sensors 332 may support a combination ofsensor modules as described above and may include various sensor typesincluding LIDAR, Color (RGB), thermal (Far-IR) or other sensor typesknown to one of skill in the art. The sensor system 320 is able toreceive data signals from a combination of sensor modules, correlate thesensor data and timely process the correlated sensor data in order tomake timely decisions based thereon.

In order for autonomous vehicles to perceive their surroundingenvironment and react accordingly, a plurality of techniques may beapplied to the sensor system to collate data from the multiple sensormodules. In particular, it may be necessary to collate the data from thesensor modules for dynamic and spatial analysis/inference, which meanstheir differences are decoupled, and digital information can betransmitted, stored and computed in a way that the vehicles and itsoperating system efficiently process and act on the different sensordata. In this regard, data from the distributed sensors can bemultiplexed to provide a unified data packet and coupled via a sensorbus to a microcontroller.

FIG. 3C depicts the operation of an MCU 348 in an autonomous drivingsystem 340 utilizing sensor module 352 and bus 358 according toembodiments of the disclosure. As illustrated, an object 341 within theautonomous navigation environment is detected by one or more sensormodules 352. As previously described, the structure and type ofsensor(s) within the sensor module 352 may vary based on design and/orpreference.

The autonomous driving system 340 may support multiple configurationsand redundancies based on the number, types and locations of sensormodules 352 installed around the vehicle. Sensor modules 352 may beactivated based on the application and external conditions. For example,when an automobile is being driven on an open highway a fewer number ofsensors and/or sensor modules may be activated relative to when anautomobile is being driven within heavy traffic. Additionally, sensorsand/or sensor modules may be activated based on a particular mode inwhich an automobile is operating. For example, particular sensors may beactivated if a vehicle is operating is a pilot mode as compared to anautonomous mode. This dynamic activation of sensors is another aspect ofthe configurability of the sensor network, which allows the system to bedynamically adapted to its environment both at installation as well asduring operation.

Sensor module(s) 352 may detect an object 341 across a plurality ofsensors and separately couple their detected data signals (shown as datastreams 1 thru n) 354 to multiplexer 356. Multiplexer 356 combines thechannels of different sensed data and generates a unified data packetcorrelating the data from each of the sensors. In some embodiments, theunified data packet comprises range and reflectivity data from LIDARtransceiver, color/RGB data from a camera, temperature data from a farinfrared detector. In other embodiments, other sensor types from otherregion of electromagnetic spectrum such as acoustics, radar or sonar maybe included. One skilled in the art will recognize that the sensormodule 352 may include various combinations of sensor module(s), sensortypes and sensor configurations. The unified data packet is coupled to abus 358, which is typically serial but may also be parallel in nature.

The data from the multiple sensors and/or sensor modules may bemultiplexed and coupled via bus 358 to a microcontroller MCU 348. MCU348 interacts with an autonomous driving system control unit(hereinafter, “ADSCU”) 342 to receive the configuration and parametersfor data acquisition from sensors.

In certain embodiments, the MCU 348 may receive external conditions andinformation about the motion of the car. MCU 348 comprises dataprocessing element 349, demultiplexer 350, calibration engine 351 anddriver 353. In certain embodiments where the bus is serial, thede-multiplexer 350 receives the data serially from multiple sensormodules and uses the calibration parameter from the calibration engineto transform the data as if it is coming from a sensor (i.e., on asensor channel basis). Calibration engine 351 provides the transformsbetween different sensors and/or sensor modules. In certain examples,these transforms are initialized to factory settings and constantlyupdated over time. The data processing element 349 comprises single ormultiple embedded algorithms for computing information such as objectdetection, velocity estimation, localization to roads and external maps.Driver 353 is responsible for activating the sensors and/or sensormodules of interest, and also providing the clock triggers.

The demultiplexer 350 de-multiplexes the unified serial data packet ofsensor data and associates the data with a corresponding sensor and/orsensor module. Thereafter, this data is provided to the calibrationengine 351, which generates transform information based on calibrationparameters received from ADSCU 342. The demultiplexer 350 also receivesthe spatial transform information and integrates it with thede-multiplexed unified serial data packet of sensor data into aparticular format such as a point cloud format.

As previously noted in FIG. 3A, the ADSCU 342 may provide sensorinstructions to MCU 302. In certain embodiments, ADSCU 342 is thecomputer in the automobile and is an element manufactured into thevehicle. As shown in FIG. 3C, ADSCU 342 receives an input in the form ofa point cloud from data processing 349, a component of MCU 348. Incertain embodiments, the ADSCU 342 may generate calibration parametersmaps 343, odometer 344, and lighting conditions 345. Other embodimentsmay have other calibration parameters and utilize a different mix ofcalibration parameters. In yet other embodiments, the odometer, lightingconditions and external map may be provided to the MCU 348 from anotherdevice within the vehicle. ADSCU 342 may also generate sensorconfigurations 346 including sensor type configurations, field of view,frame rate and region of interest. The region of interest may be, forexample, a pedestrian crosswalk or a driving lane. Via a region ofinterest identification method, the autonomous driving system 340 canfilter out amounts of unwanted raw data for the actual tracking.Effectively, MCU 348 homogenizes and decouples the different types ofsensor data. With dynamic feedback from the ADSCU 342 in the form ofcalibration parameters and sensor configuration, MCU 348 can dynamicallyconfigure sensors and /or sensor modules across different configurationsand space in an autonomous automobile environment.

FIGS. 3D and 3E illustrate methods 360 and 361 for dynamicallyconfiguring multi-sensor modules across different types of sensors andspace according to embodiments of the current disclosure comprises thefollowing steps:

Receive at MCU (Driver) sensor configuration parameters and receive atMCU (calibration engine) calibration parameters from ADSCU (step 362)

Send from MCU (Driver) configuration parameters to activate selectivesingle sensor module or multi-sensor modules (step 364)

Acquire data from an object within the environment by the selectedsensor module(s) (step 366)

Multiplex the sensor data to generate a unified data packet (step 368)

Send the unified data packet via a bus to MCU (step 370)

In the MCU, de-multiplex unified data packet into homogeneous sensordata (step 372)

In the MCU, send the homogeneous sensor data to a calibration engine(step 374)

In the MCU, generate transform information in the calibration engine andsend transform information to de-multiplexer (step 376)

In the MCU, integrate, by the de-multiplexer, the homogeneous sensordata and the transform data, and send to data processing (step 378)

Generate by data processing of the MCU, a point cloud comprising thehomogeneous sensor data and the transform data and send to ADSCU (step380)

In the ADSCU, determine/adjust control of the vehicle based on the pointcloud and generate updated sensor configurations and calibrationparameters (step 382)

Repeat step 362

FIG. 3F illustrates a method 390 for updating calibration parameters ina calibration engine according to embodiments of the current disclosurecomprises the following steps:

Receive the homogeneous sensor data from de-multiplexer (step 392)

Receive calibration parameter data from ADSCU (step 394)

Update calibration parameters in calibration engine and generatetransform information (step 396)

The above description illustrates the configurability of autonomousnavigation at a system level including the activation of certain sensorsand/or sensor modules as well as the correlation of data across thesesensors and sensor modules. In another aspect of the invention, eachsensor module may be configured to operate in accordance with apreferred set of parameters.

FIG. 4A depicts sensor module 400 and FIG. 4B depicts sensor module 402from which configurable operational parameters may be defined. Thisconfigurability not only allows for FOV definition but also sensor typeconfiguration within a sensor module. Additionally, this configurabilitymay be implemented at installation or in real-time during operation ofthe system. According to various embodiments, the sensor modules may beconfigured by defining directionality of one or more sensors within thesensor module using the physical structure of the sensor or by theinclusion of directionality elements (e.g., wedges) that define adirection of a corresponding sensor. As shown in FIG. 4B, sensor module402 may comprise a plurality of sensors 406-413 that are coupledtogether in particular architecture such that a combination ofindividual sensor FOVs is stitched together to create a broader FOV ofthe sensor module. This configurability of sensor modules allows a userto effectively build unique sensor modules by combining the differentsensors into diverse architectures. The configurability is furtherenhanced by the ability to include different sensor types within thesensor module to enhance performance relative to environmentalcharacteristics in which the module operates.

A sensor module 402 has a horizontal FOV and vertical FOV thatcorresponds to the combination of sensors 406-413. The operationalcharacteristics of each sensor 406-413 within the module 402 arecombined to provide an enhanced modular FOV. These operationalcharacteristics include the directionality of a sensor, the range of asensor, the FOV of a sensor, the type of a sensor and othercharacteristics known to one of skill in the art. In certainembodiments, particular sensors within a module may be activated ordeactivated depending on the environment in which the system isoperating. In addition, particular sensors may function as redundantelements in case one or more of the sensors fails or becomes temporarilyinoperable. The FOV of the sensor module not only depends on thespecific operational characteristics of each sensor but also on themanner in which data from these sensors is correlated and combined.

FIG. 4C illustrates a specific example Lissajous scan pattern andresolution 430 based on different vertical FOVs of a sensor according toembodiments of the present disclosure. Scan 432 illustrates a verticalscan and a horizontal scan resulting from different vertical FOVconfigurations of a sensor.

The diagrams on the right side of FIG. 4C illustrate the scanresolutions for different FOVs. FIG. 4D, vFOV 434 illustrates the scanresolution with a 2.5 degree FOV. FIG. 4E, vFOV 436 illustrates the scanresolution with a 5 degree FOV. FIG. 4F, vFOV 438 illustrates the scanresolution with a 10 degree FOV. The resolution achieved with a 2.5degree FOV is twice as dense as the resolution achieved with a 5 degreeFOV. Similarly, the resolution achieved with a 5 degree FOV is twice asdense as the resolution achieved with a 10 degree FOV. This exampleillustrates the configurability of a sensor and its resultant affect onscan pattern and resolution. One skilled in the art will recognize thatnumerous patterns and resolutions may be achieved by configuring asensor in accordance with aspects of the present disclosure.

The configurability of a sensor module is further enhanced not only bythe specific operational parameters of one or more sensors therein, butthe manner in which the one or more sensors is combined within themodule. FIG. 4G illustrates an exemplary scanning pattern 440 for asensor system comprising eight sensors within a sensor module accordingto embodiments of the present disclosure. Scanning pattern 440 may beobtained using sensor module architecture 402 in which data sensedacross the eight sensors is combined to provide enhanced resolution andfield of view. Scanning pattern 440 comprises scan 446, scan 447, scan448, scan 449, scan 450, scan 451, scan 452, and scan 453 that arecorrelated and processed to generate the pattern. In this example, thetotal field of view for sensor module architecture 402 is approximately40 degrees by 120 degrees. One skilled in the art will recognize that adiverse of modular FOVs and other module performance characteristics maybe achieved by modifying the way in which sensors are coupled together,the specific parameters of the sensors and the methods in which thesensor data is correlated and analyzed.

FIG. 4H and FIG. 4I illustrate sensor module configurations 461 and 462,respectively, according to various embodiments of the invention. Theseconfigurations are intended to be exemplary and not limiting to thescope of the invention. In one embodiment, a sensor module configurationmay be a square or rectangle shape, as illustrated in configuration 461,in which individual sensor shapes are configured to provide particularoperational characteristics within the module. Configuration 461comprises two stacked sets of sensors in which physical structuresdefine a FOV for each sensor. For example, physical size anddirectionality of a sensor may provide different angular and spatialscanning characteristics that are used within the sensor module. As aresult, sensor shape and relative locations of the sensors provide aparticular scan resolution and FOV. In another configuration, a sensormodule configuration may be a wedge shape, as illustrated inconfiguration 462, in which physical wedge elements define thedirectionality of sensors within the module. These two examplesillustrate to one of skill in the art the vast number of configurablecombinations of sensors within a sensor module. In one example, thesensors are LiDAR sensors with corresponding operational characteristicsthat allow an MCU to build an enhanced scan pattern with preferredresolution. The performance of the sensor system may be further enhancedin some embodiments by the inclusion of different sensor types within asensor module.

LIDAR sensors provide unique capabilities for autonomous driving basedprimarily on the rate and accuracy at which these sensors operate. TheseLiDAR sensors create an accurate map that can be quickly andunambiguously processed to make rapid navigation decisions with minimalerror. However, certain embodiments of the present invention supportnon-LiDAR sensors that may be included within a sensor module tosupplement the LiDAR sensor data. This multi-sensor module employingdifferent types of sensors present unique challenges in the correlationof sensed data across these sensors. Different types of sensors may havedifferent rates of data collection resulting in a more difficultcorrelation across time. Additionally, different sensors that areclosely collocated within the module may be subject to parallax errorbecause data are taken from different vantage points. Accordingly, theuse of different types of sensors within a single sensor module furthercomplicates the correlation problem previously described as well asintroduces additional complexities within the data analysis and responseprocessing of the system.

Various embodiments of the invention provide a more efficient manner forsensor data correlation across diverse types of sensors by physicallycombining the different sensors within a single module package. Thismulti-sensor module employing different sensors insures that there is a1:1 correspondence between data points from the various sensors. Thesensor data stream can be presented to the autonomous systems with thevarious sensor-type data, already combined into a correlated datapacket. The autonomous system bandwidth can then be focused on the taskof navigation rather than preprocessing and correlation of the mixeddata sets.

In one embodiment, consider a LIDAR system that returns a single pointfrom the environment. This single data point is already both a distancemeasurement (range) as well as an object reflectivity measurement withactive illumination. As a further enhancement, the LIDAR detector canalso passively measure ambient light from the scene to effectivelyrender a passive grayscale value associated with each LIDAR channel. Ina real-world navigation scenario, the color of an object carriesimportant information about its relevance. For example, stop signs andstoplights are red, yellow means caution, green may mean “information”or safe to go and so forth. Providing a unified data packet in which adata point has distance, reflectivity and color provides the autonomoussystem additional immediate information on the relevance of an object inthe field of view

Another key aspect of the real world is that it is full of livingcreatures. There are generalized algorithms that attempt to classifydetected objects based on size, shape and velocity. However, faults insuch algorithms have been demonstrated and may result in errors withinthe sensor system. One key feature of most living animal creatures thatan autonomous system may encounter is that they are warm blooded andgenerally have a different temperature than their surroundingenvironment. This characteristic can make it possible to monitor thetemperature of objects with various thermal detection technologies. Witha thermal sensor incorporated into the LIDAR sensor, yet another datatype can be incorporated into the single data packet for each data pointreported by the sensor, namely the temperature of the associate object.The ability to instantly classifies the object as a living creature hasobvious benefits to rapid autonomous system decision making. The factthat the data are naturally correlated to a real physical object greatlyimproves both reaction time and certainty of object identification. Incertain embodiments, correlation of diverse sensor data may be used toderive a confidence factor of an identified object so that a processedresponse may take into account the likelihood of an object being onetype of object versus another type.

Thermal sensors provide real-time 3D thermo-spatial information,allowing for more intelligent machine vision. For example, but withoutlimitation, an array of photodetectors sensitive to long IRelectromagnetic radiation serving alongside a scanning LIDAR system cansimultaneously localize objects in a 3D environment and discriminatewarm objects (such as living beings) from other objects in aconventional automotive environment. Active-tracking system can deliverreal-time digital information (as opposed to a passive tracking systemthat delivers a trigger signal) regarding the location and temperatureof warm objects to a vehicle control system. A single detector canprovide data over a large area by implementing a fast scanningmechanism. A large and dense array of channels can providethermo-spatial data of in all directions and with high resolution.Furthermore, detectors can be arranged so that the data is bothtemporally and spatially correlated with the LiDAR channels.

One skilled in the art will recognize that numerous combinations ofsensor types may be included within a sensor module and used to improvethe performance of the sensor system. In certain examples, thesedifferent sensor types may be used to enhance the performance of a LiDARsystem and provide greater accuracy based on certain correlated aspectsof sensed data relative to LiDAR data.

FIG. 4J illustrates a sensor system 480 that supports detection of anobject 482 using different types of sensors within a sensor module 484according to various embodiments of the invention. In this example, asensor module 484 may comprise various combinations of a LiDAR sensor,thermal/far infrared radiation (IR) sensor, visible/near IR sensor aswell as other sensor types known to one of skill in the art. The sensormodule 484 receives signals from different sensor types relative to asensed object 482. The sensor data from each different type of sensor iscaptured and provided to a multiplexer 488 along corresponding channels490-494. This data may subsequently represented on a single cloud pointfor further processing.

In a specific example, sensor 484 a (Thermal/FarIR Channel) may comprisean array of photodetectors sensitive to long IR electromagneticradiation. Sensor 484 a can simultaneously localize objects in a 3Denvironment and discriminate warm objects (such as living beings) fromother objects in a conventional automotive environment. Sensor 484 b(Visible/NearIR Channel) detects RGB color characteristics of ambientlight and may also include sensors to detect other light sources such asnear infrared light. Sensor 484 d may also include a sensor for anotherregion of electromagnetic spectrum such as acoustics, radar or sonar.These sensors 484 a, 484 b and 484 d are used to supplement the LiDARsensor 484 c to provide an enhanced sensor system performance.

Data multiplexer 488 generates a unified data packet 495 representingthe correlated data from the different sensors 484 a -d in a unifieddata packet. The data is correlated in that they are acquired from thesame point in space (or nearly the same point and unified in that theyare bundled into a single data packet).

In embodiments, aspects of the present patent document may be directedto or implemented on information handling systems/computing systems. Forpurposes of this disclosure, a computing system may include anyinstrumentality or aggregate of instrumentalities operable to compute,calculate, determine, classify, process, transmit, receive, retrieve,originate, route, switch, store, display, communicate, manifest, detect,record, reproduce, handle, or utilize any form of information,intelligence, or data for business, scientific, control, or otherpurposes. For example, a computing system may be a personal computer(e.g., laptop), tablet computer, phablet, personal digital assistant(PDA), smart phone, smart watch, smart package, server (e.g., bladeserver or rack server), a network storage device, or any other suitabledevice and may vary in size, shape, performance, functionality, andprice. The computing system may include random access memory (RAM), oneor more processing resources such as a central processing unit (CPU) orhardware or software control logic, ROM, and/or other types of memory.Additional components of the computing system may include one or moredisk drives, one or more network ports for communicating with externaldevices as well as various input and output (I/O) devices, such as akeyboard, a mouse, touchscreen and/or a video display. The computingsystem may also include one or more buses operable to transmitcommunications between the various hardware components.

FIG. 5 depicts a simplified block diagram of a computingdevice/information handling system (or computing system) according toembodiments of the present disclosure. It will be understood that thefunctionalities shown for system 500 may operate to support variousembodiments of an information handling system—although it shall beunderstood that an information handling system may be differentlyconfigured and include different components.

As illustrated in FIG. 5, system 500 includes one or more centralprocessing units (CPU) 501 that provides computing resources andcontrols the computer. CPU 501 may be implemented with a microprocessoror the like and may also include one or more graphics processing units(GPU) 517 and/or a floating point coprocessor for mathematicalcomputations. System 500 may also include a system memory 502, which maybe in the form of random-access memory (RAM), read-only memory (ROM), orboth.

A number of controllers and peripheral devices may also be provided, asshown in FIG. 5. An input controller 503 represents an interface tovarious input device(s) 504, such as a keyboard, mouse, or stylus. Theremay also be a wireless controller 505, which communicates with awireless device 506. System 500 may also include a storage controller507 for interfacing with one or more storage devices 508 each of whichincludes a storage medium such as magnetic tape or disk, or an opticalmedium that might be used to record programs of instructions foroperating systems, utilities, and applications, which may includeembodiments of programs that implement various aspects of the presentinvention. Storage device(s) 508 may also be used to store processeddata or data to be processed in accordance with the invention. System500 may also include a display controller 509 for providing an interfaceto a display device 511, which may be a cathode ray tube (CRT), a thinfilm transistor (TFT) display, or other type of display. The computingsystem 500 may also include an automotive signal controller 512 forcommunicating with an automotive system 513. A communications controller514 may interface with one or more communication devices 515, whichenables system 500 to connect to remote devices through any of a varietyof networks including the Internet, a cloud resource (e.g., an Ethernetcloud, an Fiber Channel over Ethernet (FCoE)/Data Center Bridging (DCB)cloud, etc.), a local area network (LAN), a wide area network (WAN), astorage area network (SAN) or through any suitable electromagneticcarrier signals including infrared signals.

In the illustrated system, all major system components may connect to abus 516, which may represent more than one physical bus. However,various system components may or may not be in physical proximity to oneanother. For example, input data and/or output data may be remotelytransmitted from one physical location to another. In addition, programsthat implement various aspects of this invention may be accessed from aremote location (e.g., a server) over a network. Such data and/orprograms may be conveyed through any of a variety of machine-readablemedium including, but are not limited to: magnetic media such as harddisks, floppy disks, and magnetic tape; optical media such as CD-ROMsand holographic devices; magneto-optical media; and hardware devicesthat are specially configured to store or to store and execute programcode, such as application specific integrated circuits (ASICs),programmable logic devices (PLDs), flash memory devices, and ROM and RAMdevices.

Embodiments of the present invention may be encoded upon one or morenon-transitory computer-readable media with instructions for one or moreprocessors or processing units to cause steps to be performed. It shallbe noted that the one or more non-transitory computer-readable mediashall include volatile and non-volatile memory. It shall be noted thatalternative implementations are possible, including a hardwareimplementation or a software/hardware implementation.Hardware-implemented functions may be realized using ASIC(s),programmable arrays, digital signal processing circuitry, or the like.Accordingly, the “means” terms in any claims are intended to cover bothsoftware and hardware implementations. Similarly, the term“computer-readable medium or media” as used herein includes softwareand/or hardware having a program of instructions embodied thereon, or acombination thereof. With these implementation alternatives in mind, itis to be understood that the figures and accompanying descriptionprovide the functional information one skilled in the art would requireto write program code (i.e., software) and/or to fabricate circuits(i.e., hardware) to perform the processing required.

It shall be noted that embodiments of the present invention may furtherrelate to computer products with a non-transitory, tangiblecomputer-readable medium that have computer code thereon for performingvarious computer-implemented operations. The media and computer code maybe those specially designed and constructed for the purposes of thepresent invention, or they may be of the kind known or available tothose having skill in the relevant arts. Examples of tangiblecomputer-readable media include, but are not limited to: magnetic mediasuch as hard disks, floppy disks, and magnetic tape; optical media suchas CD-ROMs and holographic devices; magneto-optical media; and hardwaredevices that are specially configured to store or to store and executeprogram code, such as application specific integrated circuits (ASICs),programmable logic devices (PLDs), flash memory devices, and ROM and RAMdevices. Examples of computer code include machine code, such asproduced by a compiler, and files containing higher level code that areexecuted by a computer using an interpreter. Embodiments of the presentinvention may be implemented in whole or in part as machine-executableinstructions that may be in program modules that are executed by aprocessing device. Examples of program modules include libraries,programs, routines, objects, components, and data structures. Indistributed computing environments, program modules may be physicallylocated in settings that are local, remote, or both.

One skilled in the art will recognize no computing system or programminglanguage is critical to the practice of the present invention. Oneskilled in the art will also recognize that a number of the elementsdescribed above may be physically and/or functionally separated intosub-modules or combined together.

It will be appreciated to those skilled in the art that the precedingexamples and embodiments are exemplary and not limiting to the scope ofthe present disclosure. It is intended that all permutations,enhancements, equivalents, combinations, and improvements thereto thatare apparent to those skilled in the art upon a reading of thespecification and a study of the drawings are included within the truespirit and scope of the present disclosure. It shall also be noted thatelements of any claims may be arranged differently including havingmultiple dependencies, configurations, and combinations.

What is claimed is:
 1. A sensor module comprising: a LiDAR transceiver coupled within the sensor module to transmit an optical signal and detect a corresponding return optical signal from an object, the LiDAR transceiver coupled to transmit sensed LiDAR data on a first channel; a first non-LiDAR sensor coupled within the sensor module to detect first sensed non-LiDAR data related to the object, the non-LiDAR sensor coupled to transmit the first non-LiDAR data on a second channel; and a multiplexer coupled to the first and second channels, the multiplexer generates a unified data packet from the LiDAR data and first non-LiDAR data and transmits the unified data packet on a bus.
 2. The sensor module of claim 1 wherein the first non-LiDAR sensor is a thermal sensor.
 3. The sensor module of claim 1 wherein the first non-LiDAR sensor is a color sensor.
 4. The sensor module of claim 1 wherein the LiDAR transceiver and non-LiDAR sensor detect data along a shared axis.
 5. The sensor module of claim 1 wherein the LiDAR data and non-LiDAR data are correlated using a 1:1 correspondence between measurement data points from the LiDAR transceiver and non-LiDAR sensor.
 6. The sensor module of claim 1 further comprising a second non-LiDAR sensor coupled within the sensor module to detect second sensed non-LiDAR data related to the object, the second non-LiDAR sensor coupled to transmit the second non-LiDAR data on a third channel.
 7. The sensor module of claim 6 wherein the multiplexer is coupled to the third channel, the multiplexer generates the unified data packet from the LiDAR data, first non-LiDAR data and the second non-LiDAR data.
 8. The sensor module of claim 7 wherein the second non-LiDAR sensor is a thermal sensor.
 9. The sensor module of claim 7 wherein the second non-LiDAR sensor is a color sensor.
 10. The sensor module of claim 1 wherein the unified data packet is correlated relative to a single cloud point.
 11. A configurable LiDAR sensor module comprising: a first LiDAR transceiver coupled within the sensor module and having a first configurable field of view, the first LiDAR transceiver generates a first scan pattern based on at least one first user-defined parameter associated with the first LiDAR transceiver; a second LiDAR transceiver coupled within the sensor module and configurably located relative to the first LiDAR transceiver, the second LiDAR transceiver having a second configurable field of view and generates a second scan pattern based on at least one second user-defined parameter associated with the second LiDAR transceiver; and a controller coupled to receive the first and second scan patterns, the controller generates a module scan pattern by correlating and combining the first and second scan patterns.
 12. The configurable LiDAR sensor module of claim 11 wherein the first and second LiDAR transceivers are structurally rectangular and positioned relative to each other to generate a preferred modular field of view.
 13. The configurable LiDAR sensor module of claim 11 wherein the first and second LiDAR transceivers are wedge shaped and positioned relative to each other to generate a preferred modular field of view.
 14. The configurable LiDAR sensor module of claim 11 wherein the first and second LiDAR transceivers are located within the module to define a preferred angular and space sensor operation.
 15. A method comprising: receiving a first sensor data associated with a LiDAR transceiver, the first sensor data corresponding to an object within a field of view of the LiDAR transceiver; receiving a second sensor data associated with a non-LiDAR sensor located proximate to the LiDAR transceiver, the second sensor data corresponding to the object; correlating the first sensor data and the second sensor data; generated a unified data packet from the first sensor data, the second sensor data and correlation between the first and second data; and transmitting the unified data packet on a bus to a microcontroller.
 16. The method of claim 15 wherein the non-LiDAR sensor is a color sensor.
 17. The method of claim 15 wherein the non-LiDAR sensor is a thermal sensor.
 18. The method of claim 15 wherein the step of correlating the first and second sensor data is performed by a multiplexer within a sensor module.
 19. The method of claim 15 wherein the LiDAR sensor and non-LiDAR sensor are located within a single package associated with a sensor module.
 20. The method of claim 15 further comprising the step of activating the non-LiDAR sensor in response to a change in an environment in which the sensor module operates. 